JOBS FOR THE FUTURE 



Achieving the Dream 

Community Colleges Count 



Achieving 
the Dream' 


DEVELOPMENTAL 

EDUCATION 

INITIATIVE 

Accelerating Achievement 


ON THE ROAD TO SUCCESS 

HOW STATES COLLABORATE AND USE DATA TO 
IMPROVE STUDENT OUTCOMES 


A WORKING PAPER BY THE ACHIEVING THE DREAM CROSS-STATE DATA WORK GROUP 


FEBRUARY 2012 



Achieving the Dream 

Community Colleges Count 

Achieving the Dream, Inc. is a national nonprofit that is 
dedicated to helping more community college students, 
particularly low-income students and students of color, stay 
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excellence, Achieving the Dream is closing achievement gaps 
and accelerating student success nationwide by: 1) improving 
results at institutions, 2) influencing public policy, 3) generating 
knowledge, and 4) engaging the public. Conceived as an 
initiative in 2004 by Lumina Foundation and seven founding 
partner organizations, today, Achieving the Dream is the largest 
non-governmental reform movement for student success in 
higher education history. With 160 community colleges and 
institutions, more than 100 coaches and advisors, and 16 state 
policy teams-working throughout 30 states and the District of 
Columbia-Achieving the Dream helps 3.5 million community 
college students have a better chance of realizing greater 
economic opportunity and achieving their dreams. 
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The Developmental Education Initiative consists of 15 
Achieving the Dream community colleges that are building on 
demonstrated results to scale up developmental education 
innovations at their institutions. Six states are committed 
to advancing their Achieving the Dream state policy work in 
the developmental education realm. Managed by MDC with 
funding from the Bill & Melinda Gates Foundation and Lumina 
Foundation, the initiative aims to expand groundbreaking 
remedial education programs that experts say are key to 
dramatically boosting the college completion rates of low- 
income students and students of color. The innovations 
developed by the colleges and states participating in the 
Developmental Education Initiative will help community colleges 
understand what programs are effective in helping students 
needing developmental education succeed and how to deliver 
these results to even more students. 
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EXECUTIVE SUMMARY 

National discussions about education reform are replete with calls for more transparency, stronger 
accountability, improved outcomes, and a more efficient use of public resources. All of these goals 
require good data about student achievement-data that are more detailed, more accurate, and more 
accessible than any state currently has. 

But collecting the data is only the first step. Central to increasing student success is a common 
approach to interpreting relevant data so that institutions and policymakers in each state can 
understand the extent of specific problems, see the impact of potential solutions, and decide where 
to act. Perhaps even more important is developing the capacity to use data on student success and 
completion trends to spur discussion and improve those outcomes within institutions, states, and 
nationally. 

Enrollment is rising across our nation's community colleges, but completion rates remain untenably 
low. Reformers are focusing on the importance of using comprehensive, high-quality data on student 
progress and completion to bring about change. A core tenet of Achieving the Dream: Community 
Colleges Count has been to embed a culture of data-informed decision making on campuses and in 
state capitals. To move this work forward, a group of data experts from Achieving the Dream's state 
policy teams came together in 2005 to design and test a valuable set of intermediate measures that 
demonstrate how community college students progress-or, as is too often the case, fail to progress- 
toward a credential or further education. Most important, this Cross-State Data Work Group showed 
the predictive power of interim indicators for identifying students in great need of intervention 
early enough to help boost their chances of staying in and finishing college. The group now includes 
Arkansas, Connecticut, Florida, Hawaii, Massachusetts, North Carolina, Ohio, Oklahoma, Texas, 
Virginia, and Washington State. 

On the Road to Success presents the next step taken by the Cross-State Data Work Group: a 
comparative analysis of a comprehensive set of intermediate milestones and final measures of 
success that practitioners and policymakers can use to rethink and remake their approaches to 
increasing student achievement. Any institution in the country can use these "Benchmarks of Student 
Success"-a set of key indicators of community college student progression and completion-to frame 
strategic planning. This is one of the first times such comparative data are being reported publicly. 

The state-level data in this brief represent only part of what the members of the Data Work Group 
have collected and reported, and their cross-state conversations will continue. Perhaps an even 
more powerful impact will be derived as states turn this analytic approach inward and examine the 
comparative effectiveness of colleges within a single state. 

INTERMEDIATE MILESTONES TO TRACK STUDENT PROGRESS 

The public, practitioners, and policymakers all need answers to essential questions about higher 
education institutions and investments. The most critical are about college completion: How many 
students complete credential programs? What are the variations by program type? What are the 
variations by student characteristics (e.g., age; gender; income; academic readiness; race/ethnicity; 
enrollment status)? 
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However, information on completion tells only part of the story. Final measures of success come too late to inform 
interventions and resource allocations that have the potential to improve student performance and help more students 
stay in school and complete. For this reason, the Data Work Group took on a new strategic priority after defining final 
success measures and benchmarks: designing a set of shorter-term measures of student success that correlate with 
and appear to predict completion or transfer success. These intermediate indicators enable states and institutions to 
track student progress toward meeting critical milestones along the road to college completion. They highlight the 
points at which students increase or decrease their chances of earning a credential. This information is crucial to 
developing and sustaining policies, programs, and supports to help more students succeed. 

To create effective measures of student progress, the Data Work Group identified and defined first-year, second-year, 
and third-year milestones that research shows are predictors of student success. At the top of this list are: persistence 
from term to term and from year to year; the accumulation of credits within a particular time frame: and progress 
through developmental education and into college-level courses. After designing interim measures, the states in the 
Data Work Group began collecting cohort data on them from their community colleges. The goal was to begin analyzing 
state-level results and draw out the implications of cross-state comparisons. 

> Student Persistence: The Data Work Group states defined and collected common measures that track student 
persistence from the fall to the spring semester of their first year, as well as persistence from their first academic 
year to their second. Fall-to-spring persistence was greater than 70 percent in all Data Work Group states; it ranged 
from 71 percent in Connecticut to 77 percent in Florida and Washington. For all states, fall-to-fall persistence was 

at or below 60 percent, ranging from 51 percent in Connecticut and North Carolina to 60 percent in Florida. The 
national fall-to-fall persistence rate is 52 percent. In each state, there is at least a 15-percentage-point drop in 
persistence between fall-to-spring and fall-to-fall measures (see Figure 5 on page 12). 

> Credit Accumulation and Course Success: Students who accrue more credits faster are more likely to attain a 
postsecondary certificate or degree than students who do not meet certain credit-accumulation thresholds. One 
measure-passing 80 percent of attempted credit hours-points to momentum in the all-important first year of 
college study. On this measure, Connecticut and Oklahoma report just under half of their student cohort hitting 
this milestone. Texas and Washington students performed better, with 61 percent and 76 percent, respectively (see 
Figure 7 on page 14). 

> Progression into College Courses: Among its intermediate measures, the Data Work Group included metrics that 
track student progression through developmental education and first college-level courses in English and math. 
Between 30 and 40 percent of students in Connecticut, Florida, and Texas complete their gatekeeper math courses 
by year three, while in Virginia about 20 percent of students reach this benchmark (see Figure 9 on page 16). 

NEXT STEPS: USING DATA TO DRIVE IMPROVEMENT 

For the Data Work Group, establishing robust student data systems was the first step toward making the interpretation 
and application of postsecondary outcome data routine in policymaking. The next step was to establish a limited but 
universal set of indicators to gauge student and institutional performance and to pinpoint when students are falling 
off track. A further step is also needed: building and institutionalizing a culture of using data to inform institutional 
improvement and policymaking. States must embrace intentional strategies to guide the regular and strategic use of 
data by colleges, their faculty and staff, and state agencies. 
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States in Achieving the Dream and its Developmental Education Initiative are collaborating to design and implement 
effective, efficient, and sustainable ways to build and support a culture of using data rigorously. This work is both 
exploratory and groundbreaking as states test new ways to: make data more visible and transparent; leverage the 
experience of high-performing institutions; and integrate data into cross-institutional sharing and learning. 

> Making Data Visible and Transparent: State data on student outcomes are too often shared in uncoordinated 
and non-strategic ways, frequently in response to a legislative mandate or accountability requirement and often 
relying on out-of-date sources. In Achieving the Dream and Developmental Education Initiative states, as well as in 
other states, institutional and state researchers are seeking to make data more accessible, transparent, and usable, 
including developing data '‘dashboards" that summarize and distill complex data in formats that are actionable. 

> Leveraging the Experience of High-performing Institutions: In-state comparisons of institutional performance 
can stimulate and structure honest and important discussions about performance variations-with implications for 
policy and practice. Comparative analysis, particularly when disaggregated to identify variations in performance of 
particular student subgroups, can help set benchmarks for current "best-in-class" outcomes that institutions can 
incorporate into their goal setting and strategic plans. 

> Integrating Performance Data into Cross-institutional Sharing and Learning: Achieving the Dream and 
Developmental Education Initiative states are identifying resources and venues that can be targeted to support a 
regular process of continuous improvement analysis and planning. These efforts range from institutionalizing the 
discussion of data in routine meetings of college leaders, to sponsoring annual statewide student success summits, 
to creating peer-learning networks. 

States that take an active role in all three areas will go a long way toward building a culture that supports the rigorous 
use of data. Much work has been done to improve the collection of data and establish appropriate sets of performance 
indicators like those presented here. The next frontier in the conversation is learning how to sustain the thoughtful use 
of data to drive large-scale institutional improvement, identify questions and issues requiring deeper research, and 
provide a critical source of feedback to inform policy. 
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INTRODUCTION 

NATIONAL DISCUSSIONS ABOUT EDUCATION REFORM ARE REPLETE WITH CALLS 
FOR MORE TRANSPARENCY, STRONGER ACCOUNTABILITY, IMPROVED OUTCOMES, 

AND A MORE EFFICIENT USE OF PUBLIC RESOURCES. ALL OF THESE GOALS REQUIRE 
GOOD DATA ABOUT STUDENT ACHIEVEMENT— DATA THAT ARE MORE DETAILED, MORE 
ACCURATE, AND MORE ACCESSIBLE THAN ANY STATE CURRENTLY HAS. 

BUT COLLECTING THE DATA IS ONLY ONE STEP. CENTRAL TO INCREASING STUDENT 
SUCCESS IS A COMMON APPROACH TO INTERPRETING RELEVANT DATA SO THAT 
INSTITUTIONS AND POLICYMAKERS CAN UNDERSTAND THE EXTENT OF SPECIFIC 
PROBLEMS, SEETHE IMPACT OF POTENTIAL SOLUTIONS, AND DECIDE WHERETO ACT. 
PERHAPS EVEN MORE IMPORTANT IS DEVELOPING THE CAPACITY TO USE DATA ON 
STUDENT SUCCESS AND COMPLETION TRENDS TO SPUR DISCUSSION AND IMPROVE 
THOSE OUTCOMES WITHIN INSTITUTIONS, STATES, AND NATIONALLY. 

Enrollments are rising across our nation's community colleges, but completion 
rates remain untenably low. Reformers are focusing on the importance of using 
comprehensive, high-quality data on student progress and completion to bring 
about change. A core tenet of Achieving the Dream: Community Colleges Count 
is to embed a culture of data-informed decision making on campuses and in 
state capitals. To move this forward, data experts from Achieving the Dream's 
state policy teams came together in 2005 to develop a common language-a 
standard set of data "indicators" that practitioners and policymakers could use to 
analyze the performance of community college students. This Cross-State Data 
Work Group then designed and tested a small but valuable set of intermediate 
measures that demonstrate how community college students progress-or, as is 
too often the case, fail to progress-toward credentials or further education (see 
box "The Achieving the Dream Cross-State Data Work Group"on page 2). Its work 
showed the predictive power of interim indicators for identifying students in 
great need of intervention early enough to help boost their chances of staying in 
and finishing college. It also pointed the way for states to develop methods and 
venues for discussions about what the data mean-and how they can be at the 
heart of institutional and state strategies for continuous improvement. 
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THE ACHIEVING THE DREAM CROSS-STATE DATA WORK GROUP 


In 2006, Achieving the Dream participants from six states-Connecticut, Florida, North Carolina, Ohio, Texas, and 
Virginia-joined forces to develop, test, and pilot a better way to measure community college performance. These 
states, the original members of the Cross-State Data Work Group, identified a more useful set of student success 
measures. They also committed to collecting data from every community college in their state and sharing the results. 
This process of identifying and defining a more complete, more accurate method of assessing student success is 
detailed in Test Drive: Six States Pitot Better Ways to Measure and Compare Community College Performance (Jobs for 
the Future 2008). Jobs for the Future, which coordinates policy work in Achieving the Dream nationally, convenes the 
Cross-State Data Work Group. 

Since the publication of Test Drive, five more states have joined the Data Work Group-Arkansas, Flawaii, Massachusetts, 
Oklahoma, and Washington. Together with the original six, they have developed and tested a set of intermediate 
milestones to help states and institutions track student progress toward college completion. Failure to meet these 
milestones serves as an early warning sign that too many students are struggling and need help. 

The pioneering efforts of the Data Work Group have had a significant influence on national initiatives promoting 
efforts to improve student outcomes, including the Developmental Education Initiative, part of Achieving the Dream 
and funded by the Bill & Melinda Gates Foundation. The Data Work Group continues to meet, collaborating with both 
Achieving the Dream and the Developmental Education Initiative. 


On the Road to Success presents the next step taken by eight states in the Cross-State Data Work Group: a comparative 
analysis of a comprehensive set of intermediate milestones and final measures of success that practitioners and 
policymakers can use to rethink and remake their approaches to increasing student achievement. Any community 
college in the country can use these "Benchmarks of Student Success"-a set of key indicators of student progression 
and completion-to frame strategic planning. 

Using these milestones, this report analyzes comparable data across the eight states. The collection and reporting of 
such data across states is rare; this is one of the first public reports of such comparative data. 

We also describe the next critical step for states: creating and implementing mechanisms and strategies for using 
this powerful analytic framework and the accompanying data to drive innovation and improve student outcomes at 
community colleges in each state and across the nation. 


Good policy and good practice begin with good data. States and community colleges participating in Achieving the 
Dream have embraced the idea of using data to identify and address obstacles to increasing student success rates. To 
this end, the members of the Cross-State Data Work Group have collaborated since 2005 to devise a comprehensive 
set of student success measures in order to gauge the effectiveness of community colleges-and to inform efforts to 
significantly boost achievement and completion. Their initial work culminated in Test Drive: Six States Pilot Better Ways 
to Measure and Compare Community College Performance (Jobs for the Future 2008). Since then, the Data Work Group 
has updated its initial analysis with more recent 2003 cohort data to see if results track with 1999 data and how data 
from two more states (Oklahoma and Washington) compare with results from the initial six (Connecticut, Florida, North 
Carolina, Ohio, Texas, and Virginia). 

The Data Work Group's initial aim was to design a set of indicators that improved on the federal approach to measuring 
community college student success. Since 1997, the federal government has required all colleges to calculate and 
report annual graduation rates for first-time, full-time degree-seeking students. 1 But each institution's graduation rate 


TAKING ANOTHER 



DRIVE” 


2 


ON THE ROAD TO SUCCESS How States Collaborate and Use Data to Improve Student Outcomes 


includes only those who complete a program within 150 percent of the "normal time to completion." For community 
colleges, the rate includes only full-time students who earn a certificate or degree within three years-part-time 
students are not counted at all. This method fails to recognize the huge proportion of community college students who 
attend part time, the large number who take more than three years to finish, and students who transfer to another 
institution before earning a degree. 

Test Drive argued for using a more comprehensive definition of success-and for counting a broader cohort of students- 
giving practitioners, policymakers, and the public a more complete picture of community college performance (see 
Table /). It recommended modifying the federal Graduation Rate Survey in several ways, including adding part-time 
students to that broader cohort. It also recommended that the definition of success include transfer to a four- 
year institution without an award. Equally important was a recommendation to extend the time period for tracking 
community college students from three years to six years, given the large number of students who take more than 
three years to complete their studies. 


TABLE 1. COMPARING COMMUNITY COLLEGE PERFORMANCE 



CURRENT FEDERAL METHOD 

CROSS-STATE DATA WORK GROUP 
METHOD 

PRIOR ENROLLMENT 

Only first-time college students 

Same as the federal method 

INTENT AT THE TIME OF 
ENROLLMENT 

Only students seeking a certificate or 
degree 

Same as the federal method 

ENROLLMENT STATUS 

Only full-time students 

Full-time and part-time students 

SUCCESSFUL OUTCOMES 

Earned degree or certificate 

Earned a degree or certificate (with or 
without transfer to a four-year institution) 

or 

Transferred without an award 
or 

Still enrolled in Year Six and earned at least 
30 college credits 

TIMEFRAME 

Three years (150 percent of "normal time" 
to completion) 

Six years 

TRACKING STUDENTS WHO 
TRANSFER WITHIN THE TWO- 
YEAR COLLEGE SECTOR 

Reporting based on individual colleges. 

Does not track outcomes of students who 
transfer to another college; college reports 
them as "transferred out." 

Reporting based on statewide community 
college system. 

Tracks outcomes of students within the 
system (and therefore across community 
colleges). 

CONTROLLING FOR FACTORS 
ASSOCIATED WITH LIKELIHOOD 
OF SUCCESS 

Excludes part-time students from analysis. 

No disaggregation of results by age at initial 
enrollment. 

Disaggregation by part-time and full-time 
status. 

Disaggregation by age at initial enrollment. 
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WHYTHE DENOMINATOR MATTERS: CALCULATING SUCCESS AND 
COMPLETION RATES 


Calls for transparency and accountability related to student progress and completion raise the stakes for states and 
their educational institutions. As public attention has turned to improving completion rates for community college 
students, debates have surged about who gets counted and what data definitions are used. One of the critical 
questions in the field focuses on the definition of a "student cohort"-stated mathematically, who should be included 
in the denominator when calculating outcomes. This question cuts to the heart of the goals that colleges set for their 
students. 

The U.S. Department of Education tracks first-time, full-time, degree-seeking students. The Cross-State Data Work 
Group advocates defining the cohort as all first-time, degree-seeking students, including part-time students. 

Maryland defines its cohort in another way: Its community colleges only count students who have completed 18 credit 
hours. California and Florida use a similar method, setting the credit threshold at 12 hours. The purpose of a minimum 
credit threshold in these states is to identify the students who truly are seeking a credential and to not exclude 
those who enroll only to take a few courses. These states argue that colleges should not be held accountable for the 
performance of students whose intention is to brush up on some skills or take a course or two but move on without a 
credential. 

The counter-argument holds that this approach could artificially inflate community college success rates by excluding 
many at-risk students. Opponents of this method argue that it is important to know what happens to all enrolling 
students and to have more rather than less data on the trajectory of low-income, minority, and academically 
underprepared students, many of whom leave college before reaching the 12- or 18-hour credit threshold. 

There are valid arguments on both sides of this debate. Nevertheless, if the ultimate goal is to maximize the number 
of students who earn meaningful credentials, the cohort definition should be as broad as possible. If narrower cohorts 
exclude certain groups of students, the implications must be made clear. 

Florida's data demonstrate the impact that varying definitions of a denominator have on the size of a cohort. Including 
all first-time college students enrolled in Florida in 2002, the cohort would be 59,259 individuals. Counting only those 
students who attempted at least 12 credit hours would reduce the cohort to 45,889; including only those who had 
attempted at least 18 credit hours would reduce it to 39,478. Almost 20,000 students-one-third-would disappear. 
Consider this thought experiment: If 10,000 of these students completed, what would the completion rate be? Using the 
18-credit definition, the completion rate is about 25 percent; using the most inclusive cohort definition, the rate is only 
17 percent. 

The American Association of Community Colleges is addressing this debate as it develops a Voluntary Framework of 
Accountability for community colleges. The VFA Preliminary Technical Guide calls for a broader definition of the cohort 
for students who are academically unprepared, as well as a cohort that uses a credit threshold for students who are 
college ready. This compromise ensures that states count the most at-risk students while accurately reflecting students' 
intent. Massachusetts has decided to collect and report both measures, the "Maryland" cohort definition and the more 
expansive definition suggested by Achieving the Dream and other initiatives. 2 


UPDATED CROSS-STATE DATA ON STUDENT SUCCESS: 
REPORTING ON THE 2003 COHORT 


Using data from the 1999 cohort in the six original Data Work Group states, Test Drive reported student outcomes 
based on the expanded Achieving the Dream approach. Since 2008, all eleven states have, in varying degrees, 
collected, reported, and shared state-level aggregate data. The most recent data, reported here, compare six-year 
outcomes for community college students who enrolled in 2003 in the original states plus Oklahoma and Washington 
(see Figure 1 on page 5). The data reported by these eight states confirm the trends from the previous analysis. 
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FIGURE 1 

STUDENT SUCCESS RATES AFTER SIX YEARS 
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Figure 1 compares outcome data across the eight states after six years for students whose outcomes should be counted 
as successes: those who received an award but did not transfer; those who both received an award and transferred; and 
those who transferred without an award. Total success rates range from 24 to over 50 percent. North Carolina, Texas, 
and Washington appear to have the greatest success in getting students to achieve one of these important outcomes. 

Variation across states is smaller for students who graduated without transferring: 10 to 20 percent of the total cohort. 
Cross-state variations are greater for groups that transfer, with or without awards. This is most likely attributable 
to variations in state transfer policies. As the 1999 data also demonstrated, a greater proportion of Texas students 
transfer without an award than in Florida, while a greater proportion of Florida students transfer after receiving an 
award. In Florida, various policies create incentives to stay in school and also make transfer easier than in many states- 
for example, guaranteed admission to a four-year public institution for students who complete an Associate's degree 
and common course numbering across two-year and four-year schools. 
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This analysis supports arguments to extend the tracking period for a 
successful outcome to six years from the federal methodology of three 
years. Figure 2 captures both four-year and six-year success rates across 
all community colleges. In every state, the cumulative success rate of 
students increases from the fourth year to the sixth year, ranging from a 
two-percentage-point increase in Oklahoma and Virginia to eight-point gains 
in Connecticut, Florida, and Washington. Flowever, additional Data Work 
Group research, not presented here, concludes that the success rates plateau 
beyond six years; additional completions among those still enrolled are 
negligible. 3 


IN EVERY STATE, THE 
CUMULATIVE SUCCESS RATE 
OF STUDENTS INCREASES 
FROM THE FOURTH YEAR TO 
THE SIXTH YEAR. HOWEVER, 
THE SUCCESS RATES PLATEAU 
BEYOND SIX YEARS. 


FIGURE 2 

STUDENT SUCCESS RATES AFTER FOUR AND SIX YEARS 
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COUNTING STUDENTS WHO ARE STILL 
ENROLLED: 

SUCCESS OR NOT? 


For its pilot study, the Cross-State Data Work Group decided that states would 
collect and report on a metric that included students who were still enrolled 
in college and had earned 30 or more college credits. Valuable trends emerge 
by examining this measure. First, students who are still enrolled with 30 or 
more credits after four years have a good chance of ultimately succeeding 
within six years. Figure 3 compares the states in terms of the percentage 
of students still enrolled with 30 or more credits, as well as those who have 
achieved success after four and six years. In most of these states, about half 
the students still enrolled after four years achieve a successful outcome by 
six years. 


IN MOST OF THESE STATES, 
ABOUT HALF THE STUDENTS 
STILL ENROLLED WITH 30 
OR MORE CREDITS AFTER 
FOUR YEARS ACHIEVE A 
SUCCESSFUL OUTCOME BY 
SIX YEARS. 


FIGURE 3 

TOTAL STUDENT SUCCESS RATES AND STUDENTS STILL ENROLLED WITH 30 OR MORE COLLEGE HOURS 
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THE VALUE OF THE CROSS-STATE DATA WORK GROUP ANALYSES IN THE 
NATIONAL DEBATE 

The Data Work Group set out in 2005 to establish a more comprehensive and useful set of final outcome measures 
to gauge the success of community college students. The group created a set of metrics that more fully reflects final 
outcomes for community college students than do the federal reporting measures reguired of all states. 

Cross-State Data Work Group Benchmarks of Student Success: Final Measures 

> Award of less than an Associate's degree without transfer 

> Award of an Associate's degree or higher without transfer 

> Award of less than an Associate's degree and transfer 

> Award of an Associate's degree or higher and transfer 

> Transferred without an award 

> Total success rate (calculated from the other final measures) 

The Data Work Group's efforts have made a significant contribution to national discussions about measuring success, 
including the American Association of Community Colleges' development of a Voluntary Framework of Accountability 
and the Committee on Measuring Student Success, established as part of the 2008 reauthorization of the F-ligher 
Education Act. The metrics and approach were a precursor to the data metrics and definitions issued by the National 
Governors Association and Complete College America, which are fast becoming the accepted metrics for collection and 
reporting student data across all states (Reyna 2010). 

See the appendix for a description of the Benchmarks of Student Success and the definitions of the data elements. 


INTERMEDIATE MILESTON ES TO TRACK 
STUDENT PROGRESS 

The public, practitioners, and policymakers all need answers to essential questions about higher education institutions 
and investments. The most critical are about completion: 

> Flow many students complete credential programs? 

> Flow does completion vary by program type? 

> Flow does completion vary by student characteristic (e.g., age; gender; income; academic readiness; race/ethnicity; 
enrollment status)? 

That said, information on completion or final success tells only part of the story. Final measures of success come too 
late to inform interventions and resource allocations that have the potential to help more students stay in school and 
complete. 

For this reason, the Cross-State Data Work Group took on an additional strategic priority: designing a set of shorter- 
term measures that correlate with and appear to be predictive of completion or transfer success. These intermediate 
indicators enable states and institutions to track student progress in meeting critical milestones along the road to 
college completion. They highlight the "momentum" or loss points where students increase or decrease their chances 
of earning a credential. This information is crucial to developing and sustaining policies, programs, and supports to 
help more students succeed. 
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As the Data Work Group began identifying a small set of powerful intermediate milestones, it benefited from two 
important influences. One was the work of Peter Ewell of the National Center for Higher Education Management 
Systems, one of the first to argue for the need to collect and report data that can shed light on the varied paths that 
students take into and through postsecondary institutions. A visual depiction of Ewell's framework (see Figure 4)- 
presented to Achieving the Dream and the Ford Bridges to Opportunity initiative— illustrates several key milestones 
along a student's path to obtaining a college credential. This analysis points toward important intermediate measures 
of student success-and to the need for consistent definitions across states of these measures of progress (Ewell 2009). 
It was the starting point for the Data Work Group. 


FIGURE 4 

"MILESTONE EVENTS” IN A STUDENT ENROLLMENT PATHWAY 
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Also influencing the Data Work Group was the path-breaking collaboration 


DEGREE 


FIELD 

EARNINGS 


of the Community College Research Center and the Washington State Board 


for Community & Technical Colleges to determine significant milestones 
that correlate with final success. 4 This research was crucial to clarifying the 


connection between a student's ability to meet certain interim measures and 
his or her likelihood of earning a credential. The Data Work Group looked 


closely at both the measures identified and the power of using state system 
data to ascertain what is happening in student progression and completion 


across a state's institutions and within each college. 


To create effective measures of student progress, the Data Work Group 
identified and defined a set of first-year, second-year, and third-year 
milestones that research shows are predictors of student success. At the 
top of this list are persistence from term to term and from year to year, the 
accumulation of credits within a particular timeframe, and progress through 
developmental education and into college-level courses. 

To select specific milestones, the Data Work Group began with metrics whose 
explanatory power had been validated by research. For example, research has 
long demonstrated that persistence from the first semester to the second 
semester and from the first academic year to the second year are important 
indicators of successful outcomes. The Data Work Group also tested the 
predictive value of the accumulation of credits earned toward a certificate 
or degree. An analysis of Florida student data (used because they were both 
available and of high quality) suggested that full-time students who earned 


AN ANALYSIS OF FLORIDA 
STUDENT DATA (USED 
BECAUSE THEY WERE BOTH 
AVAILABLE AND OF HIGH 
QUALITY) SUGGESTED THAT 
FULL-TIME STUDENTS WHO 
EARNED 24 CREDIT HOURS 
IN THEIR FIRST YEAR HAVE 
GREATER SUCCESS RATES 
THAN THOSE WHO EARN 
FEWER CREDITS. 
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SOURCE: Ewell 2009 


THE SPECIFICATION OF INTERMEDIATE MILESTONES 


As educators, government officials, and foundation leaders have embraced the agenda of dramatically increasing 
college success and credential completion, they have come to understand the need for better data on intermediate 
measures of student progress to guide the improvement efforts of institutions, systems, and states. Such measures 
typically include the acquisition of basic skills and the completion of a specific number of credits or particular gateway 
courses. 

In Taking the Next Step, commissioned by Jobs for the Future, Jeremy Offenstein and Nancy Shulock (2010) assess the 
state of the emerging field of defining, measuring, reporting, and rewarding student progress in achieving intermediate 
measures of success. The set of measures developed by the Cross-State Data Work Group is one of a number of efforts 
that Offenstein and Shulock profile and compare, which include the increasingly widely reported National Governors 
Association/Complete College America metrics for two- and four-year institutions. 


24 credit hours in their first year have greater success rates than those who 
earn fewer credits. Twenty-four credits in Year 1 proved to be a significant 
threshold for success. The second-year threshold was 42 credit hours. Both 
credit accumulation measures are among the intermediate milestones that 
the Data Work Group selected for its Benchmarks of Student Success. 

Another priority was identifying key courses for which passing correlates 
with a significantly higher likelihood of college completion. Given the 
large percentage of community college students requiring at least some 
remediation, passing the highest-level developmental education course at 
each college becomes an important milestone. A review of the Florida data 
indicated that most students who complete developmental education do 
so within their first two years of enrollment. Based on this research, the 
Data Work Group decided to define the interim measure for developmental 
education as completing required remedial courses within two years of initial 
enrollment. 

Completing developmental education is critical; so, too, is progress into 
college-level courses. As a result, the group examined data related to 
"gatekeeper" courses in English and mathematics, those often included 
in general education requirements for an Associate's degree. The analysis 
suggests that students completing gatekeeper courses by the end of their 
third year have higher success rates than those who do not finish within that 
timeframe. 

The members of the Data Work Group invested considerable time in coming 
to agreement on the intermediate milestones to be used for tracking student 
progress and a standard definition for each term (see box "Benchmarks of 
Student Success: Intermediate Milestones" on page 11). When combined with 
the improved six-year final success measures, the intermediate milestones 
provide states and institutions with a powerful tool for understanding student 
progression and completion. 5 


FOR FLORIDA, TWENTY- 
FOUR CREDITS IN YEAR 
ONE PROVED TO BE A 
SIGNIFICANT THRESHOLD 
FOR SUCCESS. THE SECOND- 
YEAR THRESHOLD WAS 
42 CREDIT HOURS. BOTH 
CREDIT ACCUMULATION 
MEASURES ARE AMONG THE 
INTERMEDIATE MILESTONES 
INCLUDED IN THE DATA WORK 
GROUP’S BENCHMARKS OF 
STUDENT SUCCESS. 
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BENCHMARKS OF STUDENT SUCCESS: INTERMEDIATE MILESTONES 

First-year student performance benchmarks: 

> Persisted fall to spring 

> Passed 80 percent or more of attempted credit hours 

> Earned 24 or more credit hours 

Second-year and third-year student performance benchmarks: 

> Persisted fall to fall 

> Passed developmental math sequence by Year 2 

> Passed gatekeeper English or higher by Year 3 

> Passed gatekeeper math or higher by Year 3 

> Achieved the two-year hour milestone 

See the appendix for a description of the Benchmarks of Student Success and the definitions of the data elements. 


NEW CROSS-STATE DATA ON INTERMEDIATE 
MILESTONES 

After designing interim measures of student success, the Data Work Group began collecting cohort data on themes 
from the states' community colleges. The goal was to begin analyzing state-level results to draw out the implications of 
cross-state comparisons. By creating indicators with common definitions and data components, the group could make 
"apples-to-apples" comparisons. Not only are the collection and reporting of such data across states rare, but this is 
also among the first time that states have reported comparative data publicly. 

The initial cross-state analysis of intermediate milestone data was a helpful first effort to benchmark state-level 
performance on these measures. The data are presented here in three parts, comparing: 

> Student persistence from term to term and year to year; 

> Accumulation of credits; and 

> Student progression through required developmental math sequences and through gatekeeper English and 
math courses. 
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FIGURE 5 

STUDENT PERSISTENCE 
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STUDENT PERSISTENCE 


The Data Work Group states defined and collected common measures that track student persistence from the fall to the 
spring semester of the first year, as well as persistence from the first academic year to the second. Figure 5 presents 
persistence data from the eight reporting states. Fall-to-spring persistence is greater than 70 percent in all states; it 
ranges from 71 percent in Connecticut to 77 percent in Florida and Washington. For all states, fall-to-fall persistence is 
at or below 60 percent, ranging from 51 percent in Connecticut and North Carolina to 60 percent in Florida. 6 In each 
state, there is at least a 15-percentage-point drop in persistence between fall-to-spring and fall-to-fall measures. 

Flypothesizing that personal finances affect students' ability to stay in college, the Data Work Group wanted to analyze 
the potential impact of income on persistence. Unfortunately, few state data systems collect such data. The Data Work 
Group turned to data on students' receipt of federal financial aid in the form of Pell Grants, which are available only to 
low-income students. While Pell Grant status is a flawed proxy for income (because many students who are eligible for 
Pell Grants do not apply for them), it remains the best income information available across state data systems. 7 
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SOURCE: Data provided by the states and analyzed by the Cross-State Data Work Group. 


FIGURE 6 

STUDENT PERSISTENCE, DISAGGREGATED BY PELL STATUS 



I FALL TO SPRING 

™ PERSISTENCE AMONG PELL RECIPIENTS 


I FALL TO SPRING 

1 PERSISTENCE AMONG NON-PELL RECIPIENTS 


I FALL TO FALL 

- PERSISTENCE AMONG PELL RECIPIENTS 


■ FALL TO FALL 

PERSISTENCE AMONG NON-PELL RECIPIENTS 


Figure 6 disaggregates persistence data based on whether or not students 
received a Pell Grant. There is a similar drop-off between fall-to-spring and 
fall-to-fall persistence both for students who receive Pell Grants and those 
who do not. However, the relative success of Pell Grant recipients differs 
between the two measures. Regarding first-year retention from fall to spring 
students who receive Pell Grants persist at higher rates than those who do 
not, with the greatest divergence in Florida and Oklahoma. This suggests 
that financial aid may play an important role in promoting persistence for 
first-year students. However, this relationship does not hold for fall-to-fall 
persistence. The two student groups achieve similar persistence rates from 
their first-year fall to their second-year fall: in half the states, non-Pell 
recipients perform the same as or better than students who receive a grant. 


REGARDING FIRST-YEAR 
RETENTION FROM FALL TO 
SPRING, STUDENTS WHO 
RECEIVE PELL GRANTS 
PERSIST AT HIGHER RATES 
THAN THOSE WHO DO 
NOT, WITH THE GREATEST 
DIVERGENCE IN FLORIDA AND 
OKLAHOMA. 


JOBS FOR THE FUTURE 


13 


SOURCE: Data provided by the states and analyzed by the Cross-State Data Work Group. 


CREDIT ACCUMULATION AND COURSE SUCCESS 


Students who accrue more credits more quickly are more likely to complete postsecondary education and attain a 
certificate or degree than students who accumulate credits slowly. Figure 7 summarizes statewide student outcomes on 
three credit-accumulation measures: 

> Students (part time and full time) passing 80 percent or more of attempted credit hours in the first year; 

> Full-time students earning 24 hours of credit in the first year; and 

> Full-time students earning 42 hours of credit and part-time students earning 24 credits in the second year. 


The first measure points to momentum in the all-important first year of college study. On this measure of momentum, 
state results vary markedly. Connecticut and Oklahoma report that just under half of their student cohort hit this 
milestone. Texas and Washington students perform better, with 61 percent and 76 percent, respectively. 

FIGURE 7 
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SOURCE: Data provided by the states and analyzed by the Cross-State Data Work Group. 


There is similar variation across the states in terms of the other credit-accumulation measures. 

Except in Connecticut and Texas, a higher percentage of students reach the first-year threshold than the second-year 
milestone. Ohio is particularly interesting: While over 50 percent of Ohio students reach the first-year credit threshold, 
only 28 percent achieve the second-year credit milestone. Washington had a less dramatic but similar drop-off. Such 
variation raises questions for further analysis, questions that require discussion and debate on the role of policy and of 
performance variations among different population subgroups. 

To explore this point, Figure 8 disaggregates by age the percentage of students in each state who pass 80 percent 
of attempted hours in the first year. Some counterintuitive trends emerge. Conventional wisdom suggests that older 
students would be more "at risk" and should have more difficulty. However, in all states but Texas, older students reach 
this success benchmark at a higher rate than younger students. There is also significant variation among states. For 
example, over 75 percent of older students meet this benchmark in Washington, compared with Oklahoma, where 


FIGURE 8 

STUDENTS PASSING AT LEAST 80 PERCENT OF ATTEMPTED HOURS, DISAGGREGATED BY AGE 
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SOURCE: Data provided by the states and analyzed by the Cross-State Data Work Group. 


slightly more than 50 percent hit this mark. While we cannot ascertain from the data itself what contributes to these 
differences, one possible reason is the role of variations in state and institutional policies governing the dropping of 
courses. 


PROGRESSION INTO COLLEGE COURSES 


Among its intermediate measures, the Cross-State Data Work Group included metrics that track student progression 
through developmental education and college-level courses in English and math. Figure 9 compares the cohorts 
of students in four states where these data are available. In Connecticut and Florida, a much higher percentage of 
students complete the developmental education sequence by their second year than in Texas and Virginia. Between 30 
and 40 percent of students in Connecticut, Florida, and Texas complete the gatekeeper math course by Year 3, while in 
Virginia about 20 percent of students reach this benchmark. 

FIGURE 9 

STUDENT PROGRESSION TOWARD COLLEGE-LEVEL COURSES 
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SOURCE: Data provided by the states and analyzed by the Cross-State Data Work Group. 


This initial analysis should raise eyebrows in Texas and Virginia. Are the data suspect? In Texas, how could so few 
students pass the developmental math sequence by the end of Year 2, and then so many pass a gatekeeper math 
course by the end of Year 3? In Virginia, what can explain the apparent low rate of math progress compared to students 
in other states? 

Because poor math skills are a major impediment for so many students, we examined student progress on the 
gatekeeper math measure more closely. Figure 10 provides a snapshot of the percentage of students passing 
gatekeeper math by Year 3, disaggregated by race. For the most part, students in the racial subgroups perform as 
previous research about achievement gaps would suggest: African-American and Latino students reach the gatekeeper 
math milestone at lower rates than other students in most states. Flowever, Florida's Latino students perform as well as 
white students. Florida college leaders might seek to find out why this is the case. Is it something about the population 
enrolled in Florida colleges or about the level of Latino performance in K-12? What interventions or policies should be 
explored further? 


FIGURE 10 

STUDENTS PASSING GATEKEEPER MATH BY YEAR 3, DISAGGREGATED BY RACE 
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DISAGGREGATING STUDENTS BY LEVEL OF 
REMEDIAL NEED 


Another important analysis of state data differentiates among students by 
level of college readiness upon enrollment in college, and the Data Work 
Group advocates that state data systems disaggregate students by this 
measure. The approach uses developmental education placement test scores 
and placement results to group students by their levels of need. Students 
who score above state-set "cut scores" are considered college ready; those 
who place into upper-level developmental courses constitute a second group; 
those who place in lower-level developmental courses constitute a third 
group. 8 

Unfortunately, not all states collect placement test data in their data 
systems. Of the Data Work Group members, only five do so at the state level— 
Connecticut, Florida, North Carolina, Texas, and Virginia. Others are beginning 
to collect this information. 


When coupled with data on intermediate milestones, disaggregation by 
remedial need gives states and institutions a much better sense of which 
students are dropping off the path toward a credential-and at which 
points. Using Texas data, Figure 11 (on page 19 ) demonstrates the power 
of disaggregating students across all other intermediate measures by 
their level of remedial need. College-ready students hit all of the identified 
milestones at higher rates than students who place into upper- or lower-level 
developmental math. 


This disparity is more pronounced on some measures than others. For 
example, the fall-to-fall and fall-to-spring persistence rates of students in 
either level of developmental math are lower than those of college-ready 
students-but only modestly so. In contrast, nearly 65 percent of college- 
ready students in Texas complete gatekeeper math by the third year, while 
only 30 percent of students placing into upper-level developmental math 
reach this milestone; fewer than 20 percent of lower-level developmental 
math students hit that mark. 

Although these results are not surprising, presentation of the data in this 
fashion highlights the magnitude of the problem. The conclusion becomes 
clear, and Texas is highlighted only as an example: to reach their college- 
completion goals, all states must figure out how to get more academically 
underprepared students to achieve key intermediate milestones. 


NEARLY 65 PERCENT OF 
COLLEGE-READY STUDENTS 
IN TEXAS COMPLETE 
GATEKEEPER MATH BY THE 
THIRD YEAR, WHILE ONLY 
30 PERCENT OF STUDENTS 
PLACING INTO UPPER- 
LEVEL DEVELOPMENTAL 
MATH REACH THIS 
MILESTONE; FEWER THAN 
20 PERCENT OF LOWER- 
LEVEL DEVELOPMENTAL MATH 
STUDENTS HITTHAT MARK. 


These data represent only part of what the states in the Data Work Group 
have collected and reported, and their cross-state conversations continue. 
Perhaps even more powerful impacts will come as they turn their analytic 
approach inward, comparing the effectiveness of colleges within a single 
state. The Developmental Education Initiative states-Connecticut, Florida, 
North Carolina, Ohio, Texas, and Virginia-have committed to undertaking this 
task. 
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FIGURE 11 

MILESTONES FORTEXAS STUDENTS, BY LEVEL OF REMEDIAL NEED 
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NEXT STEPS: 

USING THE DATA TO DRIVE IMPROVEMENT 


Since the launch of Achieving the Dream in 2004, its state policy teams have devoted considerable effort to improving 
their states' data systems and capability for analyzing data to increase student achievement. The first publication 
of the Cross-State Data Work Group -Power Tools: Designing State Community College Data and Performance 
Measurement Systems to Increase Student Success-focused sguarely on establishing these foundational, longitudinal, 
student-data systems, a prereguisite for a data-driven improvement process (Jobs for the Future 2007). In recent 
years, significant progress has been made on building stronger longitudinal data systems, within and beyond Achieving 
the Dream, thanks to the efforts of the Data Quality Campaign and other initiatives, as well as to federal funding 
through the State Longitudinal Data Grant program. 
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SOURCE: Data provided by the states and analyzed by the Cross-State Data Work Group. 


Establishing robust student-data systems is the first step toward making 
the interpretation and application of postsecondary outcomes data in 
policymaking routine. The next step is to establish a limited but universal set 
of indicators to gauge student and institutional performance and pinpoint 
when students are falling off track. The Data Work Group has devoted much 
of its recent effort to developing the set of measures described in this brief- 
measures now in use at both the state and institutional levels throughout 
Achieving the Dream and the Developmental Education Initiative. As noted, 
the National Governors Association, Complete College America, and the 
Voluntary Framework of Accountability for two-year institutions have taken 
this work to the national level (Reyna 2010). These efforts are moving quickly 
toward agreement on common intermediate and final success measures that 
state policymakers and institutional practitioners can use as they seek to 
drive improvement in student success. 


A further step is also needed: building and institutionalizing a culture of using 
data to inform institutional improvement and policymaking. Robust data 
systems and clear, common performance measures will not by themselves 
establish a statewide culture that supports data-driven improvement. States 
must embrace intentional strategies to guide the regular and strategic use of 
data by both colleges and state agencies. This new frontier is consistent with 
data-driven improvement strategies across other industries, such as health 
care and logistics. The national conversation about the use of data to drive 
improvement across fields is dynamic and exciting, and is just starting to 
emerge in the postsecondary education sector (see box “Action Analytics"). 


A FURTHER STEP IS ALSO 
NEEDED: BUILDING AND 
INSTITUTIONALIZING A 
CULTURE OF USING DATA 
TO INFORM INSTITUTIONAL 
IMPROVEMENT AND 
POLICYMAKING. 


ACTION ANALYTICS: 

SIMPLIFYING THE USE OF DATA FOR IMPROVEMENT AND PRODUCTIVITY 

Analytics are the processes of data assessment and analysis that make it possible to measure, improve, and compare 
the performance of individuals, programs, departments, institutions, enterprises, groups of organizations, or entire 
industries. Leaders in higher education use analytics to understand the performance of K-12 education, workforce 
organizations, transitions between learning and work and back again, and related issues. 

The term "action analytics" refers to analytic capacities and practices that are powerful, immediate, and lead to 
outcomes that are useful to a variety of stakeholders (Norris et al. 2008). Most important, action analytics require a 
genuine commitment to measuring and improving key aspects of productivity, innovation, and performance. The tools 
and processes of this kind of analysis can be used in reimagining academic and administrative policies and practices. 
They can help identify areas for improvement and reveal new routes to higher productivity and performance. A growing 
number of organizations, firms, and governmental systems are developing tools and services to help institutions, their 
leaders, and their staff use data to drive powerful, cost-effective improvement. 
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Achieving the Dream and Developmental Education Initiative states are collaborating to design and implement 
effective, efficient, and sustainable ways to build and support a culture of using data rigorously. This work is both 
exploratory and groundbreaking as states test new ways to: 

> Make data more visible and transparent; 

> Leverage the experience of high-performing institutions; and 

> Integrate data into cross-institutional sharing and learning. 

MAKING DATA VISIBLE AND TRANSPARENT 


Too often, states release data on student outcomes in uncoordinated and non-strategic ways, frequently in response 
to a legislative mandate or accountability requirement and often relying on out-of-date sources. Institutional and 
student data reports are frequently geared toward compliance more than strategies for action and improvement. In 
addition, inconsistent data definitions, metrics, and presentation formats contribute to confusion. Gaps in the data, 
limited capacity to present them in easy-to-use formats, and poor communication between data reporters and strategic 
planners are typical obstacles to using this critical information on student outcomes effectively. 

Achieving the Dream and Developmental Education Initiative states are tackling the challenge of making collected 
data more accessible, transparent, and usable. They are experimenting with user-friendly reporting formats that can 
be adapted to the needs of particular audiences. For example, Florida routinely provides data to multiple audiences 
through the state Department of Education's Fast Fact series, which succinctly compares community college 
performance on select measures. Following Florida's lead, the Virginia Community College System publishes Success 
Snapshots. North Carolina is revamping its annual Critical Success Factors report to focus on a small number of 
measures that are particularly meaningful to policymakers and the public regarding the effectiveness and efficiency of 
the system and individual institutions. 

Through Achieving the Dream and the Developmental Education Initiative, state and institutional researchers are 
developing data "dashboards" that summarize and distill complex data in formats that are actionable. Florida, Ohio, 
and other states are exploring ways to make data clear and visually compelling to a variety of audiences. Florida has 
invested in a partnership with the for-profit company SAS to develop an interactive online platform for the display and 
manipulation of community college institutional data by college and state leaders and faculty. 

This is an area ripe for significant progress, and important work also is taking place outside Achieving the Dream 
and the Developmental Education Initiative. For example, the Institute for Evidence-Based Change is working with 
key stakeholders throughout the educational pipeline in nine states to use data in ways that will improve student 
achievement . 9 


LEVERAGING THE EXPERIENCE OF HIGH-PERFORMING INSTITUTIONS 


In-state comparisons of institutional performance can stimulate and structure honest and important discussions about 
performance variations-and the implications for policy and practice. With this comparative data, state policymakers 
can spur conversations between and among college leaders and practitioners. 

Comparative analysis, particularly when disaggregated to identify variations in performance of particular student 
subgroups, can help set benchmarks for current best-in-class outcomes. For example, some institutions perform 
particularly well with adult students, while others have more success with English language learners or African- 
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American males. Some institutions have figured out how to get better results for students who place into upper-level 
developmental education, while others do better advancing those who place into the lowest level. Institutions can 
incorporate such findings into their goal setting and strategic planning. 

State-level actors (e.g., higher education departments, community college system offices, associations of colleges) 
are in a position to establish procedures, rules, and assurances for comparing the performance of their institutions. 
They can formalize a benchmarking process that compares results for students as a core component of continuous 
improvement. Done right, this process can be the basis for dispassionate inquiry and planning for improvement, rather 
than an exercise in finger pointing. 

Developmental Education Initiative states are exploring how to take comparative effectiveness analysis further. 

They are exploring ways to test hypotheses about why certain schools outperform others-and how state systems 
might accelerate the identification of practices and policies associated with the success of the highest-performing 
institutions. If structured carefully, this comprehensive process can lead to pivotal conversations about what a state 
can do to help all institutions achieve the results of these high performers. Such conversations can be instrumental in 
persuading institutions to abandon less effective practices in favor of those associated with superior results. 

In 2010, Jobs for the Future commissioned BTW Informing Change to study three Florida colleges that are consistently 
high performers in helping developmental math students persist, move into credit programs, and complete degrees. 
BTW tested research-based hypotheses as to why these institutions did so well between 2002 and 2008, particularly 
with developmental math students over 22 years of age upon enrollment. The resulting report can stimulate in-state 
and cross-state discussions of the implications for policy and practice (Ajose, Bhatt, & Kaur 2011). 10 This approach 
holds potential for providing tools that any state can use to further its capacity to drive and support institutional 
improvement. 


INTEGRATING PERFORMANCE DATA INTO CROSS-INSTITUTIONAL 
SHARING AND LEARNING 


In a difficult fiscal environment, how can states institutionalize discussions about data-driven improvement and conduct 
strategic planning? Achieving the Dream and Developmental Education Initiative states are identifying resources that 
can support a regular process of continuous-improvement analysis and planning. For example, they are identifying 
existing venues for sharing and disseminating data with college leaders and practitioners. Many states have routine 
meetings of presidents, chief academic officers, vice presidents of student services, and other peer-level convenings 
of staff. Other states have established annual conferences or summits that focus on improving student outcomes 
and sharing best practices. And several states are moving to create "innovators networks" that can provide an 
intensive, focused means of sharing successes and challenges and of supporting institutions that are taking the lead in 
redesigning programs and policies to improve results. 

Each of these venues presents an opportunity for states to share the kinds of data described throughout this brief. 

Flow a state can support and create incentives for participation in continuous-improvement learning networks is one 
important direction for Achieving the Dream and the Developmental Education Initiative in 2011 and 2012. 

The most common venues are the regular meetings of college leaders (e.g., councils of presidents, provosts, vice 
presidents). These individuals are often familiar with data regarding their own institutions, but they rarely view 
comparative data on other colleges. Routine gatherings of college leaders are an efficient mechanism for distilling and 
discussing the implications of this comparative information. 
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Many Achieving the Dream states have established annual conferences to elevate issues related to student success. 

For example, statewide summits in Massachusetts, Pennsylvania, and Washington bring together practitioners ranging 
from college presidents to faculty, as well as policymakers and other constituents. These convenings often include 
presentations from national experts and discussions among practitioners and policymakers about promising practices. 
They are opportunities to disseminate performance data and engage a wide range of college constituencies in 
meaningful dialogue about trends and implications. Thoughtful approaches to embedding data in these settings would 
further cement the use of this information. 

Several states are creating or expanding peer-learning networks to connect practitioners more freguently around 
common problems and promising interventions. For example, Michigan and Arkansas have each created a Center 
for Student Success as a catalyst for improving outcomes across all their community colleges. Texas is exploring the 
creation of a network of college innovators to leverage state investments in reforming the delivery of developmental 
education. These emerging state networks will incorporate performance data as a means of identifying what works and 
where additional inguiry is needed. Another promising example of cross-institutional cooperation is developing in Ohio, 
where the five colleges that have participated in Achieving the Dream and the Developmental Education Initiative serve 
as mentors for regional conversations, conveying how they use data to drive improved outcomes on their campuses. 

If states take an active role in making data more transparent, considering thoughtful processes for benchmarking 
institutional performance, and creating meaningful venues to spur cross-institutional conversations, they will go a 
long way toward building a culture that supports the rigorous use of data. Much work has been done to improve the 
collection of data and establish appropriate sets of performance indicators like those presented here. The next frontier 
in the conversation is sustaining the thoughtful use of data to drive large-scale institutional improvement, identify 
questions and issues requiring deeper research, and provide a critical source of feedback to inform policy. 


APPENDIX: 

BENCHMARKS FOR STUDENT SUCCESS 

The Cross-State Data Work Group identified and defined intermediate and final benchmarks for student success. These 
are the metrics the states have used in comparative analyses of community colleges on improving student progress and 
success, including those reported in this paper. 

FIRST-YEAR STUDENT PERFORMANCE 


BENCHMARK DESCRIPTION 


Persisted fall to spring 

The number (percent) of the original cohort who were still enrolled in the spring 
immediately following the fall starting date. 

Passed 80% or more of attempted 
hours 

The number (percent) of the original cohort who passed at least 80% of the hours in 
which they were officially enrolled during the first year. Courses of all levels are included. 
There is no minimum number of courses that must be passed. 

Earned 24 or more hours 

The number (percent) of the original cohort who passed at least 24 hours during the 
tracking period. These hours may be either in developmental education and/or college- 
level work. 

Earned a certificate prior to 24 
hours 

The number (percent) of the original cohort who earned a certificate before earning 24 
hours during the tracking period. 

Transferred prior to 24 hours 

The number (percent) of the original cohort who transferred to another institution of 
higher education before earning 24 hours during the tracking period. 


SECOND-YEAR AND THIRD-YEAR STUDENT PERFORMANCE 


BENCHMARK DESCRIPTION 


BENCHMARK 

DESCRIPTION 

Persisted fall to fall 

The number (percent) of the original cohort who were still enrolled in the fall of the year 
immediately following the fall starting date. This looks at only the two fall terms. 

Passed developmental math 
sequence by Year 2 

The number (percent) of the original cohort who completed all required developmental 
math courses by the end of the second year of tracking. This means they passed the 
highest-level course required. 

Passed gatekeeper English or 
higher by Year 3 

The number (percent) of the original cohort who completed the gatekeeper English course 
or a higher-level English course by the end of the third year of tracking. Success is based 
upon any enrollment in the course and does not have to be the initial enrollment. 

Passed gatekeeper math or higher 
by Year 3 

The number (percent) of the original cohort who completed the gatekeeper math or a 
higher-level math course by the end of the third year of tracking. Success is based upon 
any enrollment in the course and does not have to be the initial enrollment. 

Achieved the two-year hour 
milestone 

The number (percent) of the original cohort who successfully passed the hour milestones 
during the tracking period. These hours may be either in developmental education or 
college-level work. The number of hours required depends upon the enrollment status of 
the students during their first fall term only, not their current status. The two-year hour 
milestone is 24 hours for part-time students and 42 hours for full-time students. 
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FINAL MEASURES 


BENCHMARK DESCRIPTION 


Award of less than Associate's 

The number (percent) of the original cohort who have been granted an award below the 

degree without transfer 

Associate's level as recorded in the state database and have NOT transferred. This award 
is usually called a certificate. 

Award of Associate's degree or 
higher without transfer 

The number (percent) of the original cohort who have been granted an Associate's degree 
and have not transferred. This includes all Associate's degrees (e.g., Associate in Arts, 
Associate in Science, Associate in Applied Science). 

Award of less than an Associate's 

The number (percent) of the original cohort who have been granted an award below the 

degree and transferred 

Associate's level as recorded in the state database and have transferred to a two- or 
four-year degree program. This award usually is called a certificate. Students who have 
transferred without a degree and then have returned to earn a degree will also be counted 
in this category. 

Award of Associate's degree or 
higher and transferred 

The number (percent) of the original cohort who have been granted an Associate's degree 
and have transferred. This includes all Associate's degrees, no matter their individual 
titles, (e.g., Associate in Arts, Associate in Science, Associate in Applied Science). 

Transferred without an award 

The number (percent) of the original cohort who have transferred to a four-year 
institution without an award within the community college system. This may include 
transferring to the upper division of a community college offering four-year degrees. 

Still enrolled with 30 or more 

The number (percent) of the original cohort who are still enrolled in the community 

college hours 

college system in the final year of tracking (fall, spring, or summer, depending on the 
state's defined end point) and have earned at least 30 college-level hours. 

Total success rate-this 

The sum of the percents for earned award below Associate's degree (with and without 

benchmark is calculated from 

transfer), earned Associate's degree (with and without transfer), transferred without an 

other benchmarks 

award, and still enrolled with 30 or more college hours. 


BENCHMARK DATA ELEMENT DEFINITIONS 

CORE DATA 


DATA ELEMENT DEFINITION 


DATA ELEMENT 

DEFINITION 

Original cohort 

All beginning fall award-seeking students. This includes students who are seeking 
short-term certificates, longer-term certificates, and Associate's degrees, both full and 
part time. This is the "IPEDS fall cohort," with the addition of the part-time students. 
This group will form the denominator for most of the percentage calculations. In a few 
cases, selected subgroups of students will be subtracted from this denominator prior to 
calculating the percentages. 

Award-seeking student 

Either the student or the institution has indicated that the student is seeking an 
Associate's degree or a certificate (as defined by the state). 

Full-time/part-time status 

The definition of full time is the criteria used by IPEDS. Students enrolled in at least 
four-fifths of a full course load their first term-usually 12 credits out of a possible 15 
credits-are considered full time. All other students are considered part time. This initial 
enrollment status is used to classify the student and remains the same, whether or not the 
student's actual enrollment status changes at some later point. 
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Traditional age 

The student's age when they first entered (e.g., the beginning fall term). This can be 
the age reported by the institution or the difference between the entering year and 
the student's birth year. Students 22 or younger are classified as traditional age. This 
classification does not change during the tracking period. 

Older 

The student's age when they first entered (e.g., the beginning fall term). Students over 22 
are classified as "older." This classification does not change during the tracking period. 

College ready 

The student is allowed to enroll in college-level courses in all areas beginning with his or 
her entering term. Students were not required to take remedial coursework based upon 
their placement test scores. States that do not collect placement test scores should use 
only the "all students" line on the form. In states without uniform standards for "college 
ready," this classification does not change during the tracking period. 

Developmental education 

Students who needed to take developmental education as indicated by their placement 
test scores for those states that collect that information. States that do not collect 
this information should not attempt to break out these students and should use the all 
students line on the form. 

Spring 

For the "Persisted fall to spring" benchmark, spring is the term following the beginning fall 
term. 

Fall 

For the "Persisted fall to fall" benchmark, the second fall is the fall term of the year 
following initial enrollment. 

Persisted 

The student was listed in the enrollment records of the appropriate spring or fall term. The 
student does not have to pass any courses, merely return to school. 

Percent 

The number of students from the original cohort who achieved the benchmark under 
consideration, divided by the total number in the original cohort (or corrected cohort, if 
appropriate). 

Earned x hours within y years 

The subset of the cohort that passed x hours within y years. The number of hours varies 
by the initial enrollment status of the students and by the time period involved. Except for 
the final benchmark of "Still enrolled with 30 or more college hours," the hours are a sum 
of both developmental and college-level hours. The amount of tracking time also varies 
depending upon the benchmark being used. 

Passed 

To pass a course, the student earned a grade of D or better. In the case of gatekeeper 
courses, "passed" is defined by the state. 

Officially enrolled 

The student is listed as being enrolled in a given course on the database used for this 
project. 

Gatekeeper or higher course 

The gatekeeper course is usually the first college-level course that is required for an 
Associate's degree. It is often a part of the general education requirements. FHigher 
courses are those taken following completion of the gatekeeper. In measuring success 
rates for gatekeeper math, only college transfer degree-seeking students are counted. 

Transfer 

The student was found in the enrollment file of the state's four-year institutions or was 
found in the enrollment files of four-year institutions that report to the National Student 
Clearinghouse. If the state has community colleges that offer four-year degrees, the 
upper-division files of those institutions may also be examined and any students found 
included. If a student is enrolled at two institutions at the same time, the student will be 
counted at the first institution they started in. 
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Tracking period 

The amount of time the cohort is followed. This time may vary for different data runs, and 
begins with a fall term and ends with the final term in a reporting year. This may be either 
spring or summer, depending upon the state involved. Cohorts will not be measured past 
year six. 

SUPPLEMENTAL DATA 


DATA ELEMENT 

DEFINITION 

Developmental education English 
only 

Those students who need only developmental English or writing but not developmental 
math. 

Developmental math breakdown 

States should classify students who test into development math as upper level (more 
moderate need) or lower level (more substantial need) by first calculating the midpoint of 
the score range used on the exam(s) to place students into developmental math, and then 
classifying students who scored above the midpoint as upper-level and those who scored 
at or below the midpoint as lower-level. For example, Florida places students who score 
between 20 and 71 on the elementary algebra ACCUPLACER exam into developmental 
math. To further divide these students into upper-level and lower-level developmental 
math, one would calculate the midpoint score in that range of 20 to 71, which is 45.5 (71- 
20=51. 51/2=25.5. 25.5+20=45.5) and classify all students who scored above that (i.e. 46 
to 71) as upper level and those who scored below that (i.e., 20 to 45) as lower level. States 
should note the actual score ranges they used for classifying students as either upper or 
lower in the footnote reference included on the template. States, like Texas, that have a 
method for classifying students as upper- or lower-level developmental math (e.g., score 
below one standard deviation) should use their existing method and again footnote the 
score ranges used. For students who are not required to take developmental math (i.e., 
technical programs), a math equivalent should be measured for these students. 

Low income-Pell Grant received 

Students receiving a Pell Grant. This is not all students who are eligible but rather those 
who actually receive the award. States also can use other applicable measures that may 
be standard within their definition of low-income. 

Not low-income 

All students in the original cohort who did not receive a Pell Grant. 

Race/ethnicity 

The race/ethnicity reported by the institution for the student. This will transition to be 
consistent with revised questions to be introduced by the U.S. Census Bureau in the next 
few years. 

Gender 

The gender reported by the institution for the student. 


JOBS FOR THE FUTURE 


27 



ENDNOTES 


1 The Integrated Postsecondary Education Data System (IPEDS) added graduation rates (GRS) as an annual component in 1997 to help institutions 
satisfy the requirements of the Student Right-to-Know legislation. Data are collected on the number of students entering the institution as first-time, 
full-time, degree/certificate-seeking undergraduate students in a particular year (cohort), by race/ethnicity and gender; the number completing their 
program within 150 percent of normal time to completion; and the number that transfer to other institutions if transfer is part of the institution's 
mission. See: http://nces.ed.gov/ipeds/glossary 

2 For more information on the Voluntary Framework of Accountability, see: http://www.aacc.nche.edu/Resources/aaccprograms/vfa/Pages/default.aspx 

3 See Test Drive (JFF 2008). 

4 Washington was seeking measures that could be the foundation of a revised performance-based funding system. 

5 See also Jenkins 2011. 

6 The national fall-to-fall persistence rate for community college students was 52 percent in 2008, according to the National Center for Education 
Statistics. 

7 For a discussion of these issues and an experimental alternative using Census data, see the research by the Community College Research Center 
(Crosta 2006). 

8 The authors recognize that cut scores are a blunt measure and caution is advised, but for these purposes they provide another useful way to parse 
and analyze the data. 

9 See: http://www.iebcnow.org and http://www.calpass.org 

10 For more information on BTW, see: http://btw.informingchange.com; for the full report, see http://www.jff.org/publications/education/high-flyers- 
policies-and-strategies-boos/1301 
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